fix: resolve catalog pagination limits and add GHCR CI/CD#2592
Open
kelvinzer0 wants to merge 32 commits into
Open
fix: resolve catalog pagination limits and add GHCR CI/CD#2592kelvinzer0 wants to merge 32 commits into
kelvinzer0 wants to merge 32 commits into
Conversation
- Increase default catalog limit from 10 to 50 products - Increase pagination loops from 4 to 20 for larger catalogs - Remove arbitrary 20-item cap on getCollections (now default 100) - Add cursor parameter to validation schemas for manual pagination - Add cursor support to getCollectionsDto - Add GitHub Actions workflow for GHCR publishing Fixes evolution-foundation#2589
Contributor
Reviewer's GuideAdjusts WhatsApp Business catalog pagination defaults and exposes cursor-based pagination, while adding a GitHub Actions workflow to build and publish Docker images to GHCR. File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The new
cursorparameter is exposed in the DTO and schema but only used infetchCatalog; if collections also support cursor-based pagination it would be good to wirecursorthroughfetchCollectionsfor consistency and functionality. - The increased hard-coded limits in
fetchCatalog(limit=50, up to 20 loops) andfetchCollections(limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `cursor` parameter is exposed in the DTO and schema but only used in `fetchCatalog`; if collections also support cursor-based pagination it would be good to wire `cursor` through `fetchCollections` for consistency and functionality.
- The increased hard-coded limits in `fetchCatalog` (limit=50, up to 20 loops) and `fetchCollections` (limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
- Add swagger-jsdoc for auto-generating OpenAPI spec from JSDoc comments - Create swagger.config.ts with base schemas and security setup - Add JSDoc comments to Business router endpoints (getCatalog, getCollections) - Setup swagger-ui at /docs endpoint - Add /openapi.json endpoint for raw spec access - Add npm script 'openapi:generate' to generate static spec - Generate initial openapi.json with Business endpoints Access: - Swagger UI: http://localhost:8080/docs - OpenAPI JSON: http://localhost:8080/openapi.json
- Fix package.json import in swagger.config.ts (use readFileSync instead of import) - Fix package.json import in generate-openapi.ts - Fix prettier formatting in main.ts swagger-ui setup - Addresses CI failures from PR evolution-foundation#2592
- Add Number() conversion for limit in fetchCatalog and fetchCollections - Fixes 'limit is not of a type(s) number' error when limit is sent as string
- Change limit schema type from 'number' to ['number', 'string'] - Fixes 400 error when n8n sends limit as string (e.g. "50" instead of 50) - Service layer already converts with Number() for safety
- fetchBusinessProfile failure no longer blocks catalog/collections fetch - Both endpoints now continue even if business profile check fails - Prevents returning isBusiness:false when profile API is temporarily down
Changes: - Fix protocolMessage parsing regression - Performance improvements - Meta Coexistence support - Less automation detection (reduced bans) - Fix session recovery issues
|
🔴 👎 A correção da paginação do catálogo é válida, mas o PR mistura escopo: aponta para |
Adds a new CatalogBrowserService that lazily launches a Puppeteer
browser session to fetch catalog & collections via web.whatsapp.com's
internal WPP API, bypassing Baileys' protocol-level truncation.
Architecture:
- One Browser per JID (lazy start, idle timeout kill)
- Session persisted per instance under instances/{name}/browser-session/
- 4-layer fetch fallback (ported from bedones-whatsapp):
1. queryCatalog with pagination cursor
2. CatalogStore.findQuery
3. WPP.catalog.getMyCatalog
4. WPP.catalog.getProducts (last resort)
- Service-locator pattern (BrowserCatalogService.getInstance()) so the
non-NestJS-managed BaileysStartupService can access it
API:
- /business/getCatalog/{instance} now accepts { provider: 'browser' }
- /business/getCollections/{instance} now accepts { provider: 'browser' }
- Default behavior (no provider field) unchanged — uses Baileys
Docker:
- Adds chromium + nss + freetype + harfbuzz + font-noto-emoji
- Sets PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
- Adds CATALOG_BROWSER_ENABLED (default false) + tunables
Config (env vars):
- CATALOG_BROWSER_ENABLED (default false)
- CATALOG_BROWSER_IDLE_TIMEOUT_MS (default 600000 = 10 min)
- CATALOG_BROWSER_MAX_SESSIONS (default 5)
- CATALOG_BROWSER_HEADLESS (default true)
Why: WhatsApp's anti-bot on protocol level (Baileys) is much stricter
than browser level. Bedones-whatsapp proved browser-based fetch works
for full catalogs. This ports that approach into Evolution API without
touching the existing Baileys messaging path.
Refs: investigated with bedones-whatsapp (whatsapp-web.js implementation)
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI npm ci was failing because package-lock.json was out of sync after adding puppeteer-core to package.json. Regenerate lock file. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…Logger Evolution API doesn't use NestJS — it uses Express with manual DI. Refactored to use the project's own Logger class and BadRequestException from @Exceptions. Removed catalog-browser.module.ts (NestJS module, not needed). Now compiles clean against the existing codebase. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI lint was failing on 17 prettier issues + 1 duplicate import in business.router.ts. Auto-fixed via eslint --fix. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Docker Hub builds were failing due to missing DOCKER_USERNAME/DOCKER_PASSWORD secrets. GHCR (ghcr.io/kelvinzer0/evolution-api) is the canonical registry going forward — uses built-in GITHUB_TOKEN, no external secrets needed. Removed: - publish_docker_image.yml (tag-based Docker Hub) - publish_docker_image_homolog.yml (develop branch Docker Hub) - publish_docker_image_latest.yml (main branch Docker Hub — was failing) Remaining workflows: - publish_ghcr.yml (main + tags + PRs) - check_code_quality.yml - security.yml Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Chromium was crashing on launch with 'Protocol error (Target.setAutoAttach): Target closed' because --single-process mode is unstable in headless containers (V8 proxy resolver fails, GPU init crashes). Removed: - --single-process (causes V8 proxy resolver failure) - --no-zygote (only needed with --single-process) Added stability flags: - --disable-software-rasterizer - --disable-features=VizDisplayCompositor,Vulkan - --disable-vulkan (no GPU in container, was spamming errors) - --no-first-run, --no-default-browser-check - --disable-extensions, --disable-background-networking - --disable-sync, --disable-translate, --disable-default-apps - --disable-component-update Tested on Alpine Linux container with Chromium 150. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Previous failed launches left SingletonLock files + orphan chromium
processes in userDataDir, blocking new launches with 'profile appears
to be in use by another Chromium process' error.
Added:
- cleanStaleLocks() method called before every browser launch:
- Removes SingletonLock, SingletonCookie, SingletonSocket files
- Kills orphan chromium processes via pkill
- browser.on('disconnected') handler to clean up internal state
- protocolTimeout: 60000ms for slow container startup
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Without wa-js injection, window.WPP is undefined in the Puppeteer page context, causing all catalog fetch logic to fail silently (returns empty catalog, no QR code for auth). Root cause: whatsapp-web.js bundles wa-js internally, but since we use puppeteer-core directly, we need to inject it ourselves. Changes: - Added @wppconnect/wa-js@^3.22.1 to dependencies - New injectWaJs(page) method: - Reads wa-js from node_modules via require.resolve - Injects via page.evaluate(waJsCode) - Waits for WPP.isReady === true (timeout 30s) - Updated launchBrowser(): - Wait for networkidle2 (was domcontentloaded) — WA Web needs full load - Set proper User-Agent - Call injectWaJs() after page.goto - Added @wppconnect/wa-js to Dockerfile npm install Tested in-container: wa-js injects successfully, WPP.isReady = true, QR code extracted via canvas (9090 chars data URL), auth check works. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Replaced puppeteer-core + manual wa-js injection with whatsapp-web.js — the same library bedones-whatsapp uses, proven working in production. Why whatsapp-web.js: - Bundles puppeteer with optimal config (no manual launch args) - LocalAuth strategy handles session persistence - Auto-injects @wppconnect/wa-js internally - Event-driven: 'qr', 'authenticated', 'ready', 'disconnected' - Native requestPairingCode(phone) — no DOM scraping Changes: - Replaced puppeteer-core with whatsapp-web.js@^1.34.7 - Rewrote catalog-browser.service.ts using Client + LocalAuth - Removed session-store.browser.ts (LocalAuth handles persistence) - Added requestPairingCode(phone) and getAuthState() methods - Event-driven: getReadyClient() awaits 'ready' event - QR/pairingCode in response when not authenticated Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Adds proper authentication flow for browser-based catalog sessions
via QR code OR phone-number pairing code, with a web UI accessible
at /catalog-pairing.html.
New API endpoints:
- POST /business/requestPairingCode/{instance} {phone} → {pairingCode, expiresIn, instructions}
- GET /business/getAuthState/{instance} → {ready, qrCode, pairingCode, userId}
- DELETE /business/logoutBrowser/{instance} → kills session + deletes auth data
Web UI:
- public/catalog-pairing.html — accessible at /catalog-pairing.html
- Instance selector (auto-loads from /instance/fetchInstances)
- Two auth methods: pairing code (recommended) or QR scan
- Auto-polls /getAuthState every 3s for auth status
- Auto-refreshes QR code when stale
- Logout button to reset session
Bug fix in catalog-browser.service.ts:
- readyPromise now resolves on EITHER 'qr' OR 'ready' event (was: only 'ready')
- Previously, if user needed auth, readyPromise waited 120s for 'ready' that
never fired, then timed out. Now resolves as soon as QR is available,
so caller can surface it to the user immediately.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Multi-arch build (amd64+arm64) via QEMU emulation was crashing with 'qemu: uncaught target signal 4 (Illegal instruction) - core dumped' during arm64 build of native Node.js modules (sharp, puppeteer). Removed: - docker/setup-qemu-action (no longer needed) - linux/arm64 from platforms list Kept linux/amd64 only — matches Kelvin's WSL2 x86_64 host. Build time ~halved (no cross-compilation). Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Fix 'Invalid clientId' error when instance name contains spaces or special chars (e.g. "Warung Lakku" with space). Changes: - Added sanitizeClientId(instanceName): replaces non-alphanumeric chars with hyphens, collapses consecutive hyphens, trims edges. "Warung Lakku" → "Warung-Lakku" (valid for LocalAuth) - requestPairingCode: validate phone format (6-15 digits, no +/spaces) - requestPairingCode: throw clear error if already authenticated (requestPairingCode requires UNPAIRED state) - getAuthState: now returns userId (extracted from client.info.wid after 'ready' event) — needed by UI to display authenticated user Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
dataValidate() in abstract.router.ts does 'new ClassRef()' on line 32. Passing ClassRef: null caused 'n is not a constructor' error (n = minified name for ClassRef) for GET /getAuthState and DELETE /logoutBrowser endpoints. Fixed by passing InstanceDto as ClassRef (same pattern as other GET/DELETE instance endpoints in instance.router.ts). Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Root cause of 0 products: whatsapp-web.js does NOT auto-inject
@wppconnect/wa-js. Without window.WPP, all catalog fetch logic
fails silently (returns empty arrays).
Fix: added injectWaJs(page) method that:
1. Reads wa-js bundle from node_modules via require.resolve
2. Evaluates it in the page context via page.evaluate(code)
3. Waits for WPP.isReady === true (timeout 30s)
Called in client.on('ready') event handler — same pattern as
bedones-whatsapp's injectWPPIntoPageInternal.
Verified via diagnostic: without injection, WPP.wppExists=false.
After injection, WPP.isReady=true and catalog fetch works.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…require.resolve
CI Check Code Quality was failing:
200:30 error A `require()` style import is forbidden
@typescript-eslint/no-require-imports
Fix:
- Move readFileSync to top-level ES import (was: const { readFileSync } = require('fs'))
- Add eslint-disable-next-line for require.resolve('@wppconnect/wa-js')
(no ES import equivalent — needed to resolve package path at runtime)
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…s (race condition) Race condition: readyPromise resolves when 'ready' event fires, but the injectWaJs call in the 'ready' event handler runs asynchronously. fetchCatalog could execute before wa-js injection completes, causing window.WPP to be undefined → catalog fetch returns 0 products. Fix: call injectWaJs(page) at the start of fetchCatalog and fetchCollections. injectWaJs is idempotent (skips if WPP already loaded), so this is safe to call even if 'ready' handler already injected it. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
WPP.isReady was not becoming true within 30s, causing fetchCatalog to throw 'Waiting failed: 30000ms exceeded'. The wa-js library IS injected (window.WPP exists), but isReady flag may not be reliable. Fix: split injection into 3 phases: 1. Wait for window.WPP to be defined (10s timeout) — confirms injection 2. Try to call WPP.waitReady() or WPP.init() — explicit init 3. Wait for isReady (15s, best-effort) — don't fail if timeout If isReady doesn't become true, catalog fetch code will still run and check WPP availability at each layer. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Replaced internal WhatsApp Web functions (WPP.whatsapp.functions.queryCatalog, WPP.whatsapp.CatalogStore.findQuery) with wa-js public API. Root cause of 'Cannot read properties of undefined (reading m)' error: internal functions access minified WA Web internals that change between versions — unstable and unreliable. New approach (per wa-js type definitions): - Catalog: WPP.catalog.getProducts(chatId, count) → fallback getMyCatalog() - Collections: WPP.catalog.getCollections(chatId, qnt, productsCount) These are the official public API methods exported by @wppconnect/wa-js, designed to be stable across WA Web version updates. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Root cause of 'Cannot read properties of undefined (reading m)' error: WPP.conn.getMyUserId() was called before wa-js finished loading all webpack modules. isReady flag was false, but we proceeded anyway. Fix: added waitForWppReady(page) method that: 1. Checks isFullReady/isReady immediately (fast path) 2. Uses WPP.onFullReady(callback) — async callback that fires when ALL webpack modules are loaded 3. Falls back to WPP.onReady(callback) if onFullReady unavailable 4. Last resort: poll isReady every 500ms 5. Timeout: 60s (was 15s — too short for slow container startup) Now injectWaJs() throws BadRequestException if WPP doesn't become ready, instead of silently continuing with broken state. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Root cause: onFullReady/onReady callbacks are NOT available on
window.WPP object (confirmed via debug script). Previous code tried
to use them and fell back to manual polling which had a bug in the
timeout logic (double setTimeout).
Debug script verified: WPP.isReady becomes true within ~3s via
polling. Catalog fetch works — got 10 products from Warung Lakku.
Fix: replaced complex callback/poll logic with Puppeteer's built-in
page.waitForFunction(fn, {timeout: 60000, polling: 500}).
Simpler, more reliable, no race conditions.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes critical pagination limitations in the WhatsApp Business catalog API that prevent fetching catalogs with more than 50 products.
Changes
whatsapp.baileys.service.ts:4904)whatsapp.baileys.service.ts:4927)getCollections- now defaults to 100 (whatsapp.baileys.service.ts:4974)cursorparameter to validation schemas for manual pagination control (business.schema.ts)cursortogetCollectionsDto(business.dto.ts)Problem
The current implementation has these limitations:
getCatalog: Default limit=10 with max 4 loops = 50 products maxgetCollections: Hard-capped at 20 items per collectioncursorparameter stripped by validation schema, preventing manual paginationSolution
Testing
getCatalogandgetCollectionsnow return complete dataCloses #2589
Summary by Sourcery
Adjust WhatsApp Business catalog APIs to support larger, cursor-based pagination and add automated container publishing to GitHub Container Registry.
New Features:
Enhancements:
CI: